This invention relates generally to management of a distributed cluster of computers, and more particularly to processes for efficiently updating the computers in a distributed cluster of machines in a large distributed processing system.
Computers are typically required to be upgraded (updated) on a regular basis for a variety of different reasons. Upgrades may be for the purpose of improving performance, correcting software bugs and security problems, adding new functions and capabilities, and up grading system components, to name a few. The updates may include, for example, new application software, new operating system software, new hardware drivers, hardware configuration changes, firmware updates, system configuration file changes or any other file system changes on the machine.
Generally, upgrading the software on a computer is a lengthy time-consuming process which is rendered more difficult because of the large number of different types of updates which typically must be preformed. In a distributed processing system comprising a distributed cluster of computers, the processing workload is shared among the computers of the cluster, and it is necessary that these computers of the cluster be updated together in parallel so that they are compatible with one another, and so that they have the same or equivalent functionality and processing capabilities. Since the machines that need to be updated have user data on them, a complete re-imagining of a machine is not possible because user data must be preserved. When dealing with a large cluster of machines, the upgrade process may be slower on some machines than on others. Thus, while the upgrade process should be done at the same time on all machines, the upgrade of each machine is preferably done independently, and, for performance reasons, without any communication to a controlling master server.
Furthermore, since multiple reboots of a machine may be necessary during upgrade, the machine should continue to update itself automatically through the reboots, as required. Also, the update of some machines of a large cluster may fail partway through the upgrade process due, for instance, to faults or to unexpected inconsistencies introduced into specific machines. When a failure occurs, it is desirable to be able to resume the upgrade process on the failed machine without the necessity of redoing actions that have already been successfully completed. In order to be reliable and scalable, the upgrade process must be able to track successfully completed actions. However, it cannot rely upon centralized technologies, such as NFS, which would introduce a single point of failure into the upgrade process and adversely impact the ability of the process to scale. One, well known way to track actions which have been completed and uncompleted on the various machines of a cluster is to use a log file to record actions. However, a log file is inefficient to maintain, error-prone, and intolerant of failures such as the log file being lost or the occurrence of unexpected interventions that put the system into an unknown state.
It is desirable to provide new and improved methods for upgrading the machines in a distributed computing cluster that address the foregoing and other known problems of upgrading machines in large clusters, and which are efficient, reliable and scalable. It is to these ends that the present invention is directed.